Automated vehicle repair estimation by aggregate ensembling of multiple artificial intelligence functions

ABSTRACT

Automated vehicle repair estimation by aggregate ensembling of multiple artificial intelligence functions is provided. A method comprises receiving a plurality of vehicle repair recommendation sets for a damaged vehicle, wherein each of the vehicle repair recommendation sets identifies at least one recommended vehicle repair operation of a plurality of the vehicle repair operations for the damaged vehicle; aggregating a plurality of the recommended vehicle repair operations; generating a composite vehicle repair recommendation set that identifies the aggregated recommended vehicle repair operations; and providing the composite vehicle repair recommendation set to one or more vehicle repair insurance claims management systems.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application claims priority to U.S. Provisional PatentApplication No. 62/908,348, filed Sep. 30, 2019, entitled “Methods ForProcessing Data From An Artificial Intelligence Model And DevicesThereof”; U.S. Provisional Patent Application No. 62/908,354, filed Sep.30, 2019, entitled “Methods For Processing Data From An ArtificialIntelligence Model Using Voting Technique And Devices Thereof”; and U.S.Provisional Patent Application No. 62/908,361, filed Sep. 30, 2019,entitled “Methods For Processing Data From An Artificial IntelligenceModel Using Preferential Technique And Devices Thereof”; the disclosuresthereof incorporated by reference herein in its entirety.

DESCRIPTION OF RELATED ART

The disclosed technology relates generally to artificial intelligence(AI), and more particularly some embodiments relate to the use of invehicle repair estimation.

SUMMARY

A claimed solution rooted in computer technology overcomes problemsspecifically arising in the realm of computer technology.

In general, one aspect disclosed features a system, comprising: ahardware processor; and a non-transitory machine-readable storage mediumencoded with instructions executable by the hardware processor toperform a method comprising: receiving a plurality of vehicle repairrecommendation sets for a damaged vehicle, wherein each of the vehiclerepair recommendation sets identifies at least one recommended vehiclerepair operation of a plurality of the vehicle repair operations for thedamaged vehicle; aggregating a plurality of the recommended vehiclerepair operations; generating a composite vehicle repair recommendationset that identifies the aggregated recommended vehicle repairoperations; and providing the composite vehicle repair recommendationset to one or more vehicle repair insurance claims management systems.

Embodiments of the system may include one or more of the followingfeatures. In some embodiments, each of the vehicle repair recommendationsets identifies a plurality of images of the damaged vehicle; and themethod further comprises: selecting one or more of the images of thedamaged vehicle, and identifying the selected one or more of the imagesin the generated composite vehicle repair recommendation set. In someembodiments, each of the vehicle repair recommendation sets identifies ascore and/or confidence percentage for the recommended vehicle repairoperation; and the method further comprises: identifying the scores forthe recommended vehicle repair operations in the generated compositevehicle repair recommendation set. In some embodiments, each of thevehicle repair recommendation sets includes one or more images of thedamaged vehicle; selecting a plurality of the recommended vehicle repairoperations comprises: providing the vehicle repair recommendation setsto an artificial intelligence function; and the artificial intelligencefunction is trained using a plurality of vehicle repair training sets,wherein each vehicle repair training set comprises: one or more imagesof a further damaged vehicle, and a composite vehicle repairrecommendation set for the further damaged vehicle. In some embodiments,each of the vehicle repair recommendation sets is generated by arespective further artificial intelligence function. In someembodiments, each of the further artificial intelligence functions istrained. In some embodiments, the method further comprises requestingone or more of the further artificial intelligence functions bere-trained when a predetermined event occurs. In some embodiments,aggregating a plurality of the recommended vehicle repair operationscomprises: aggregating the plurality of the recommended vehicle repairoperations based on statistical aggregation methodologies comprising atleast one of mean, max, min, variance, and standard deviation.

In general, one aspect disclosed features a non-transitorymachine-readable storage medium encoded with instructions executable bya hardware processor of a computing component, the machine-readablestorage medium comprising instructions to cause the hardware processorto perform a method comprising: receiving a plurality of vehicle repairrecommendation sets for a damaged vehicle, wherein each of the vehiclerepair recommendation sets identifies at least one recommended vehiclerepair operation of a plurality of the vehicle repair operations for thedamaged vehicle; aggregating a plurality of the recommended vehiclerepair operations; generating a composite vehicle repair recommendationset that identifies the aggregated recommended vehicle repairoperations; and providing the composite vehicle repair recommendationset to one or more vehicle repair insurance claims management systems.

Embodiments of the non-transitory machine-readable storage medium mayinclude one or more of the following features. In some embodiments, eachof the vehicle repair recommendation sets identifies a plurality ofimages of the damaged vehicle; and the method further comprises:selecting one or more of the images of the damaged vehicle, andidentifying the selected one or more of the images in the generatedcomposite vehicle repair recommendation set. In some embodiments, eachof the vehicle repair recommendation sets identifies a score and/orconfidence percentage for the recommended vehicle repair operation; andthe method further comprises: identifying the scores for the recommendedvehicle repair operations in the generated composite vehicle repairrecommendation set. In some embodiments, each of the vehicle repairrecommendation sets includes one or more images of the damaged vehicle;selecting a plurality of the recommended vehicle repair operationscomprises: providing the vehicle repair recommendation sets to anartificial intelligence function; and the artificial intelligencefunction is trained using a plurality of vehicle repair training sets,wherein each vehicle repair training set comprises: one or more imagesof a further damaged vehicle, and a composite vehicle repairrecommendation set for the further damaged vehicle. In some embodiments,each of the vehicle repair recommendation sets is generated by arespective further artificial intelligence function. In someembodiments, each of the further artificial intelligence functions istrained. In some embodiments, the method further comprises requestingone or more of the further artificial intelligence functions bere-trained when a predetermined event occurs. In some embodiments,aggregating a plurality of the recommended vehicle repair operationscomprises: aggregating the plurality of the recommended vehicle repairoperations based on statistical aggregation methodologies comprising atleast one of mean, max, min, variance, and standard deviation.

In general, one aspect disclosed features a method comprising: receivinga plurality of vehicle repair recommendation sets for a damaged vehicle,wherein each of the vehicle repair recommendation sets identifies atleast one recommended vehicle repair operation of a plurality of thevehicle repair operations for the damaged vehicle; aggregating aplurality of the recommended vehicle repair operations; generating acomposite vehicle repair recommendation set that identifies theaggregated recommended vehicle repair operations; and providing thecomposite vehicle repair recommendation set to one or more vehiclerepair insurance claims management systems.

Embodiments of the method may include one or more of the followingfeatures. In some embodiments, each of the vehicle repair recommendationsets identifies a plurality of images of the damaged vehicle; and themethod further comprises: selecting one or more of the images of thedamaged vehicle, and identifying the selected one or more of the imagesin the generated composite vehicle repair recommendation set. In someembodiments, each of the vehicle repair recommendation sets identifies ascore and/or confidence percentage for the recommended vehicle repairoperation; and the method further comprises: identifying the scores forthe recommended vehicle repair operations in the generated compositevehicle repair recommendation set. In some embodiments, each of thevehicle repair recommendation sets includes one or more images of thedamaged vehicle; selecting a plurality of the recommended vehicle repairoperations comprises: providing the vehicle repair recommendation setsto an artificial intelligence function; and the artificial intelligencefunction is trained using a plurality of vehicle repair training sets,wherein each vehicle repair training set comprises: one or more imagesof a further damaged vehicle, and a composite vehicle repairrecommendation set for the further damaged vehicle. In some embodiments,each of the vehicle repair recommendation sets is generated by arespective further artificial intelligence function. In someembodiments, each of the further artificial intelligence functions istrained. In some embodiments, the method further comprises requestingone or more of the further artificial intelligence functions bere-trained when a predetermined event occurs. In some embodiments,aggregating a plurality of the recommended vehicle repair operationscomprises: aggregating the plurality of the recommended vehicle repairoperations based on statistical aggregation methodologies comprising atleast one of mean, max, min, variance, and standard deviation.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure, in accordance with one or more variousembodiments, is described in detail with reference to the followingfigures. The figures are provided for purposes of illustration only andmerely depict typical or example embodiments.

FIG. 1 illustrates a vehicle estimate automation system according tosome embodiments of the disclosed technology.

FIG. 2 illustrates a process for vehicle repair workflow according tosome embodiments of the disclosed technology.

FIG. 3 illustrates a vehicle repair estimate automation system accordingto some embodiments of the disclosed technology.

FIG. 4 illustrates a vehicle repair estimate automation processaccording to some embodiments of the disclosed technology.

FIG. 5 illustrates an aggregate ensembling process according to someembodiments of the disclosed technology.

FIG. 6 illustrates an example operation according to the aggregateensembling process of FIG. 5 .

FIG. 7 illustrates a voting ensembling process according to someembodiments of the disclosed technology.

FIG. 8 illustrates an example operation according to the votingensembling process of FIG. 7 .

FIG. 9 illustrates a preferential ensembling process according to someembodiments of the disclosed technology.

FIG. 10 illustrates an example operation according to the preferentialensembling process of FIG. 9 .

FIGS. 11A, B illustrate a random ensembling process according to someembodiments of the disclosed technology.

FIG. 12 illustrates an example operation according to the randomensembling process of FIG. 11 .

FIGS. 13A, B illustrate an adaptive ensembling process according to someembodiments of the disclosed technology.

FIG. 14 illustrates an example operation at a time T_(t) according tothe adaptive ensembling process of FIG. 13 .

FIG. 15 illustrates an example operation after relearning the weights,at a time T_(t+m+2), according to the adaptive ensembling process ofFIG. 13 .

FIG. 16 is an example computing component that may be used to implementvarious features of embodiments described in the present disclosure.

The figures are not exhaustive and do not limit the present disclosureto the precise form disclosed.

DETAILED DESCRIPTION

With the advent of high-power, cost effective computing systems came theincreased automation of numerous facets of our contemporary society. Inthe insurance and other casualty and loss industries, for example,computerized claims estimating, processing, tracking and payment systemshave long been in use to streamline processes and to expedite claimshandling and closure.

Advances in Artificial Intelligence (AI) have enabled the use of AI toassist in the estimating process. However, AI implementations may differmarkedly. For example, given the same input, two different AI functionsmay produce quite different estimates. Embodiments of the disclosedtechnology exploit these variations in AI implementations to harness thepower of multiple AI functions in generating a vehicle repair estimate.In various embodiments, a claim package is provided to a plurality of AIfunctions in various ways, and the estimates output by the AI functionsare combined in various ways.

Some embodiments employ an aggregating approach. According to theseembodiments, the same claim package is provided to each of the AIfunctions, and the estimates produced by the AI functions areaggregated.

Some embodiments employ a voting approach. According to theseembodiments, the same claim package is provided to each of the AIfunctions, and the estimates produced by the AI functions are combinedaccording to scores assigned to the estimates by the AI functions.

Some embodiments employ a preferential approach. According to theseembodiments, the same claim package is provided to each of the AIfunctions, and each of the AI functions is assigned a respective rank.The estimates produced by the AI functions are selected according to theranks.

Some embodiments employ a random approach. According to theseembodiments, different portions of the claim package are provided to theAI functions in a random manner, and the estimates produced by the AIfunctions are combined.

Some embodiments employ an adaptive approach. According to theseembodiments, weights are learned for each operation for each componentfor each AI function. The same claim package is provided to each of theAI functions, and the estimates produced by the AI functions areselected according to the learned weights.

In this description, various embodiments are disclosed for vehiclerepair estimate workflow automation. However, embodiments of thedisclosed technology apply to other processes as well. For example,embodiments may apply to generating estimates for medical procedures,and the like. These and other applications will be apparent to oneskilled in the relevant art after reading this description. Beforedescribing embodiments of the disclosed technology in detail, it isuseful to describe an example environment in which the disclosedtechnology may be implemented.

FIG. 1 illustrates a vehicle estimate automation system 100 according tosome embodiments of the disclosed technology. Multiple users may beinvolved in the vehicle repair workflow. For example, referring to FIG.1 , the users may include the insured 112, a claims adjuster 114, arepairer 116 such as an employee of a repair shop, an independentappraiser 118, and the like. Each user may employ a respective clientdevice 122, 124, 126, 128. Each client device may be implemented as adesktop computer, laptop computer, smart phone, smart glasses, embeddedcomputers and displays, diagnostic devices and the like.

The system 100 may include a vehicle repair estimate automation tool102, which may be implemented as one or more software packages executingon one or more server computers 104. Each user may employ a respectiveclient device 122, 124, 126, 128 to access the tool 102 over a network130 such as the Internet.

The tool 102 may include a plurality of AI functions 108. The AIfunctions 108 may be implemented in any manner. For example, one or moreof the AI functions 108 may be implemented as trained machine learningmodels. The machine learning models may include computer vision machinelearning models, natural language processing machine learning models,and the like. The system 100 may include one or more databases 106,which may store vehicle repair procedures, completed estimates,estimates in process, data regarding parts, part costs, labor, laborcosts, and the like. The databases 106 may include one or more naturallanguage processing (NLP) techniques. The natural language processingdatabases may include rules, documents, and the like for use with thenatural language processing machine learning models. In someembodiments, a NLP machine learning model may be trained with vehiclerepair content. The vehicle repair content may include vehiclespecifications, vehicle repair procedures, position statements, partscatalogs, and the like. The NLP machine learning model may ingest andprocess the vehicle repair content to generate machine learning rulesand processed NLP content databases. The NLP machine learning model mayfurther curate the ingested vehicle repair content through otherartificial intelligence technologies including image analysis, textmining, deep analysis, and the like.

The system 100 may include a claims management system 140. The claimsmanagement system 140 may be operated, for example, by a vehicleinsurer.

FIG. 2 illustrates a process 200 for vehicle repair workflow accordingto some embodiments of the disclosed technology. The elements of thedisclosed processes are presented in a particular order. However, itshould be understood that, in various embodiments, one or more elementsmay be performed in a different order, in parallel, or omitted.Referring to FIG. 2 , the process 200 may begin with a vehicle accident,at 202, and may continue with the vehicle owner reporting the accidentto an insurance company, and taking the vehicle to a repair facility, at204. Alternatively, the owner may take the vehicle to a repair facility,at 206, which may report the accident to the insurance company, at 208.

Next, a vehicle damage assessment is performed, at 210. For example, astaff appraiser of an insurance company may visit the damaged vehicle totake photos of the damage. Alternatively, the owner may send photos ofthe damaged vehicle to the insurance company. Next, the process mayinclude the generation of a vehicle repair estimate, at 212, asdescribed below in detail. Based on the vehicle repair estimate, therepair of the vehicle may take place, at 214. When the repair iscomplete, the repaired vehicle may be delivered to the vehicle owner, at216.

FIG. 3 illustrates a vehicle repair estimate automation system 300according to some embodiments of the disclosed technology. Elements ofthe system 300 may be implemented within the vehicle repair estimatingsystem 100 of FIG. 1 . Referring to FIG. 3 , the system 300 may includeensembling AI function 302 and a plurality of evaluating AI functions304A, B, C. While for clarity of explanation only three evaluating AIfunctions 304 are shown and discussed, it should be appreciated that anynumber of evaluating AI functions 304 may be used.

In some embodiments, one or more of the evaluating AI functions 304 maybe implemented and/or operated by the same entity that implements and/oroperates ensembling AI function 302. In some embodiments, one or more ofthe evaluating AI functions 304 may be implemented and/or operated by anentity that is different from the entity that implements and/or operatesensembling AI function 302. For example, one or more of the evaluatingAI functions 304 may be implemented and/or operated by a third-partyvendor or the like. One advantage of this arrangement is that evaluatingAI functions may be added and removed as desired.

As noted above, the AI functions 302,304 may be implemented in anymanner. The system 300 may include an applications programming interface(API) gateway (GW) 306 to provide an interface between the ensembling AIfunction 302 and other systems. In the example of FIG. 3 , one or moreclaims management systems 308 may access the ensembling AI function 302through the API GW 306.

The claims management system 308 may submit a claim package 310 to theensembling AI function 302. The claim package may include datadescribing a damaged vehicle. The data may include images such as photosof the damaged vehicle, video of the damaged vehicle, audio recordingsdescribing the damaged vehicle, text describing the damaged vehicle, andthe like.

The ensembling AI function 302 may distribute the claim package 310 tothe evaluating AI functions 304. Referring to FIG. 3 , the ensembling AIfunction 302 may distribute claim packages 310A,B,C to the evaluating AIfunctions 304A,B,C, respectively. In some embodiments, the claimpackages 310A,B,C may be the same. In other embodiments, the claimpackages 310A,B,C may differ. For example, each of the claim packages310A,B,C may be a different subset of the claim package 310.

Referring to FIG. 3 , the evaluating AI functions 304A,B,C may generaterecommendation sets 312A,B,C, respectively. The evaluating AI functions304 may be implemented in different manners, so that when given the sameclaim package 310, the evaluating AI functions 304 may generatedifferent recommendation sets 312.

A recommendation set 312 may identify at least one recommended vehiclerepair operation, of a plurality of the vehicle repair operations, forthe damaged vehicle. For example, a recommended vehicle repair operationmay indicate that a front bumper of the vehicle should be replaced. Arecommendation set 312 may identify a score for a recommended vehiclerepair operation. Each score may be a floating-point value thatindicates a projected accuracy of the recommended vehicle repairoperation, with higher floating-point values indicating greaterprojected accuracy. A recommendation set 312 may include metadataobjects of the damaged vehicle, for example including one or more of theimages, in the corresponding claim package 310. In some instances, inaddition to scores, a mathematical and/or statistical confidencepercentage may also be included to indicate confidence around AIinferred accuracy of each recommended operation. Higher confidencepercentages for an AI decision may indicate higher confidence in the AIdecision.

The ensembling AI function 302 may generate a composite recommendationset 314 based on the recommendation sets 312 generated by the evaluatingAI functions 304, as described in detail below. The compositerecommendation set 314 may identify at least one recommended vehiclerepair operation, a score for a recommended vehicle repair operation,one or more of the images in the corresponding claim package 310, andthe like. The ensembling AI function 302 may provide the compositerecommendation set 314 to one or more claims management systems 308.

FIG. 4 illustrates a vehicle repair estimate automation process 400according to some embodiments of the disclosed technology. The process400 may be performed, for example, by the vehicle repair estimatingsystems 100 and 300 of FIGS. 1 and 3 . Referring to FIG. 4 , the process400 may include receiving a claim package, at 402. In the example ofFIG. 3 , the ensembling AI function 302 may receive a claim package 310from one or more claims management systems 308 through the API GW 306.

Referring again to FIG. 4 , the process 400 may include determining aconfiguration of the vehicle repair estimate automation process 400, at404. In some embodiments, the process 400 may be configured to employone or more ensembling techniques. In the example of FIG. 4 , theensembling techniques may include aggregate ensembling, at 406, votingensembling, at 408, preferential ensembling, at 410, adaptiveensembling, at 412, and random ensembling, at 414. In the example ofFIG. 3 , the ensembling techniques may be performed by the ensembling AIfunction 302 in conjunction with the evaluating AI functions 304. Eachof these techniques is described in detail below.

Referring again to FIG. 4 , the process 400 may include providing theresulting composite recommendation set, at 420. In the example of FIG. 3, the ensembling AI function 302 may provide the compositerecommendation set 314 to one or more claims management systems 308through the API GW 306.

FIG. 5 illustrates an aggregate ensembling process 500 according to someembodiments of the disclosed technology. The process 500 may beperformed, for example, by the vehicle repair estimating systems 100 and300 of FIGS. 1 and 3 . Referring to FIG. 5 , the process 500 may includedistributing the claim package to the evaluating AI functions, at 502.In the example of FIG. 3 , ensembling AI function 302 may distributeclaim packages 310A,B,C to the evaluating AI functions 304A,B,C,respectively.

Referring again to FIG. 5 , the process 500 may include receiving avehicle repair recommendation set from the evaluating AI functions,wherein each of the vehicle repair recommendation sets identifies atleast one recommended vehicle repair operation of a plurality of thevehicle repair operations, at 504. In the example of FIG. 3 , ensemblingAI function 302 may receive recommendation sets 312A,B,C from theevaluating AI functions 304A,B,C, respectively.

Referring again to FIG. 5 , the process 500 may include aggregating aplurality of the recommended vehicle repair operations, at 506. In theexample of FIG. 3 , the ensembling AI function 302 may aggregate thereceived recommendation sets 312. In some embodiments, aggregating thereceived recommendation sets 312 may include combining two or more ofthe received recommendation sets 312. In some embodiments, aggregatingmay include aggregating the plurality of the recommended vehicle repairoperations based on statistical aggregation methodologies comprising atleast one of mean, max, min, variance, and standard deviation. In someembodiments, redundant recommended vehicle repair operations may beomitted from the resulting combination. Other aggregation techniques arecontemplated as well.

Referring again to FIG. 5 , the process 500 may include generating acomposite vehicle repair recommendation set that identifies theaggregated recommended vehicle repair operations, at 508. In the exampleof FIG. 3 , the ensembling AI function 302 may aggregate the receivedrecommendation sets to form the composite vehicle repair recommendationset 314.

Referring again to FIG. 5 , the process 500 may include providing thecomposite vehicle repair recommendation set to one or more vehiclerepair insurance claims management systems, at 510. In the example ofFIG. 3 , the ensembling AI function 302 may provide the compositerecommendation set 314 to one or more claims management systems 308.

In some embodiments, one or more of the evaluating AI functions 304 mayinclude one or more trained AI functions. For example, an AI functionmay be implemented as a trained computer vision machine learning model.The machine learning model may be trained, for example, with images ofother damaged vehicles and the corresponding vehicle repair operationsapplied to repair those vehicles. In these embodiments, a trained AIfunction may be retrained occasionally, for example when a predeterminedretraining trigger event occurs. A retraining trigger event may be afunction a defined lapsed period of time (say, every six months, forexample), a result of a comparison between a pre-defined evaluationmetric of an AI function and a sample hold out (also known as test dataset), and may consider factors such as measuring precision, recall andF1 score (accuracy). When an evaluation metric falls below a certainpredefined metric, retraining can be automatically retriggered prior toa lapse of a predefined period. For example, the ensembling AI function302 may compare the components of the recommendation sets received froman evaluating AI function against predetermined benchmarks, and based onthe comparison, may declare a retraining trigger event.

Referring again to FIG. 5 , the process 500 may include determiningwhether a predetermined retraining trigger event has occurred, at 512.The process 500 may include requesting one or more of the evaluatingartificial intelligence functions be re-trained when a predeterminedevent occurs, at 514. In the example of FIG. 3 , the ensembling AIfunction 302 may determine whether a predetermined retraining triggerevent has occurred, and may request one or more of the evaluatingartificial intelligence functions 304 be re-trained when thepredetermined event occurs.

FIG. 6 illustrates an example operation 600 according to the aggregateensembling process 500 of FIG. 5 . Referring to FIG. 6 , the ensemblingAI function 602 receives a claim package from a claims management system608 via an API GW 606. For clarity, only the images in the claim packageare discussed. However, it should be appreciated that the claim packagemay contain other data objects in addition to, or instead of, theimages, and that those data objects may be treated in a similar manner.

The claim package example illustrated includes five images 1,2,3,4,5 ofa damaged vehicle. The ensembling AI function 602 distributes the imagesto the evaluating AI functions 604. In this example, the ensembling AIfunction 602 distributes all of the images to each of the evaluating AIfunctions 604. The evaluating AI functions 604 process the images.

Each of the evaluating AI functions 604 returns a differentrecommendation set. The evaluating AI function 604A returns arecommendation set that includes images 1,2 and two recommendedoperations with components and scores. The legend in FIG. 6 indicatesidentifiers for components and operations. One of the recommendedoperations is LFB, OP11, 0.7, which indicates replacement of the LowerFront Bumper with a score of 0.7. The other recommended operation isUFB, OP9, 0.5, which indicates repair of the Upper Front Bumper with ascore of 0.5. But while the examples presented herein describe AIfunction inferencing outcome examples such as component, laboroperation, associated score, and the like, it should be understood thatadditional AI outcomes may include other factors, for example such asrepair labor hours, type of damage, area of damage, and the like.

The evaluating AI function 604B returns a recommendation set thatincludes images 3,4 and one recommended operation with component andscore. The recommended operation is LF, OP9, 0.6, which indicates repairof the Left Fender with a score of 0.6.

The evaluating AI function 604C returns a recommendation set thatincludes images 2,3,4 and two recommended operations with components andscores. One of the recommended operations is FB, OP9, 0.5, whichindicates repair of the front bumper with a score of 0.5. The otherrecommended operation is H, OP26, 0.4, which indicates painless dentrepair of the hood with a score of 0.4.

The ensembling AI function 602 may aggregate these recommendation setsto form the composite vehicle repair recommendation set. In thisexample, the composite recommendation set includes a union of the imagesin the recommendation sets provided by the evaluating AI functions 604,namely images 1,2,3,4. The composite recommendation set also includes aunion of the recommended operations and scores in the recommendationsets provided by the evaluating AI functions 604.

In some embodiments, the ensembling AI function 602 prefers granularity.As used herein, the term “granularity” may pertain to various minorparts of the vehicle that an AI function can identify, rather than majorpanels or sub-panels/components. In the example of FIG. 6 , theevaluating AI function 604A has provided two recommended operations forthe front bumper (LFB and UFB), while the evaluating AI function 604Chas provided only one (FB). Therefore the ensembling AI function 602includes the two recommended operations for the front bumper (LFB andUFB), while omitting the single recommendation (FB).

The ensembling AI function 602 provides the composite recommendation setto the claims management systems 608.

FIG. 7 illustrates a voting ensembling process 700 according to someembodiments of the disclosed technology. The process 700 may beperformed, for example, by the vehicle repair estimating systems 100 and300 of FIGS. 1 and 3 . Referring to FIG. 7 , the process 700 may includedistributing the claim package to the evaluating AI functions, at 702.In the example of FIG. 3 , ensembling AI function 302 may distributeclaim packages 310A,B,C to the evaluating AI functions 304A,B,C,respectively.

Referring again to FIG. 7 , the process 700 may include receiving aplurality of vehicle repair recommendation sets for a damaged vehicle,wherein each of the vehicle repair recommendation sets identifies (i) atleast one component of the damaged vehicle, (ii) a recommended vehiclerepair operation for each identified component, and (iii) a score foreach recommended vehicle repair operation, at 704. The vehicle repairrecommendation sets may include other recommended operations such asrepair labor hours, area of damage, type of damage, and the like. In theexample of FIG. 3 , each of the recommendation sets 312 generated by theevaluating AI functions 304 may include one or more recommended vehiclerepair operations. In addition, each of the recommendation sets 312 maybe associated with a component of the damaged vehicle, and may have ascore.

Referring again to FIG. 7 , the process 700 may include, when aplurality of the vehicle repair recommendation sets identify recommendedvehicle repair operations for one of the components of the damagedvehicle, selecting the recommended vehicle repair operation for the oneof the components having the highest score or highest confidencepercentage, and unselecting the other recommended vehicle repairoperations for the one of the components, at 706. In the example of FIG.3 , when multiple recommendation sets 312 include a recommended vehiclerepair operation, the ensembling AI function 302 may select therecommended vehicle repair operation having the highest score or highestconfidence percentage, and unselect the recommended vehicle repairoperation(s) having lower scores.

Referring again to FIG. 7 , the process 700 may include generating acomposite vehicle repair recommendation set, wherein the compositevehicle repair recommendation set identifies the selected recommendedvehicle repair operation, and wherein the composite vehicle repairrecommendation set does not identify the unselected recommended vehiclerepair operation, at 708. In the example of FIG. 3 , the ensembling AIfunction 302 may include, in the composite vehicle repair recommendationset 314, all of the recommended vehicle repair operations from therecommendation sets 312, except the unselected recommended vehiclerepair operation(s).

Referring again to FIG. 7 , the process 700 may include providing thecomposite vehicle repair recommendation set to one or more vehiclerepair insurance claims management systems, at 710. In the example ofFIG. 3 , the ensembling AI function 302 may provide the compositerecommendation set 314 to one or more claims management systems 308.

Referring again to FIG. 7 , the process 700 may include determiningwhether a predetermined retraining trigger event has occurred, at 712.The process 700 may include requesting one or more of the evaluatingartificial intelligence functions be re-trained when a predeterminedevent occurs, at 714. In the example of FIG. 3 , the ensembling AIfunction 302 may determine whether a predetermined retraining triggerevent has occurred, and may request one or more of the evaluatingartificial intelligence functions 304 be re-trained when thepredetermined event occurs. A retraining trigger event may be asdescribed above.

FIG. 8 illustrates an example operation 800 according to the votingensembling process 700 of FIG. 7 . Referring to FIG. 8 , the ensemblingAI function 802 receives a claim package from a claims management system808 via an API GW 806. For clarity, only the images in the claim packageare discussed. However, it should be appreciated that the claim packagemay contain other data objects in addition to, or instead of, theimages, and that those data objects may be treated in a similar manner.

The claim package includes five images 1,2,3,4,5 of a damaged vehicle.The ensembling AI function 802 distributes the images to the evaluatingAI functions 804. In this example, the ensembling AI function 802distributes all of the images to each of the evaluating AI functions804. The evaluating AI functions 804 process the images.

Each of the evaluating AI functions 804 returns a differentrecommendation set. The evaluating AI function 804A returns arecommendation set that includes images 1,2,3,4,5 and three recommendedoperations with components and scores. The legend in FIG. 8 indicatesidentifiers for components and operations. One of the recommendedoperations is LFB, OP11, 0.7, which indicates replacement of the lowerfront bumper with a score of 0.7. Another recommended operation is UFB,OP9, 0.5, which indicates repair of the upper front bumper with a scoreof 0.5. A third recommendation is LF, OP11, 0.9, which indicatesreplacement of the left fender.

The evaluating AI function 804B returns a recommendation set thatincludes images 1,2,3,4,5 and two recommended operations with componentsand scores. One recommended operation is H, OP26, 0.3, which indicates apaintless dent repair for hood. Another recommended operation is LF,OP11, 0.6, which indicates replacement of the left fender with a scoreof 0.6.

The evaluating AI function 804C returns a recommendation set thatincludes images 1,2,3,4 and three recommended operations with componentsand scores. One of the recommended operations is FB, OP9, 0.5, whichindicates repair of the front bumper with a score of 0.5. Anotherrecommended operation is H, OP26, 0.4, which indicates paintless dentrepair of the hood with a score of 0.4. Another recommended operation isLF, OP9, 0.7, which indicates a repair of the left fender.

In the example of FIG. 8 , two of the recommendation sets include arecommended repair operation for the hood. Therefore, the ensembling AIfunction 802 selects the recommended repair operation for the hoodhaving the higher score, and unselects the other. That is, theensembling AI function 802 selects the recommendation operation H, OP26,0.4, returned by the evaluating AI function 804C, and unselects therecommended operation H, OP11, 0.3 returned by the evaluating AIfunction 804C.

In the example of FIG. 8 , all three of the recommendation sets includea recommended operation for the left fender. Therefore, the ensemblingAI function 802 selects the recommended repair operation for the leftfender having the highest score, and unselects the other two. That is,the ensembling AI function 802 selects the recommendation operation LF,OP11, 0.9 returned by the evaluating AI function 804A, and unselects therecommended operation LF, OP11, 0.6 returned by the evaluating AIfunction 804B, and unselects the recommended operation LF, OP9, 0.7returned by the evaluating AI function 804C.

In some embodiments, the ensembling AI function 802 identifies the scoreof each selected operation in the composite recommendation set, as shownin FIG. 8 . In other embodiments, the ensembling AI function 802combines the score of each selected operation with the scores of thecorresponding unselected operation(s). Any technique of combining thescores may be used. For example, the scores may be averaged. In theexample of FIG. 8 , the scores for the hood operation may be averages sothat the operation H, OP26, 0.35 is provided in the compositerecommendation set. In the example of FIG. 8 , there are two unselectedoperations for the left fender, but only one has the same operation code(OP11) as the selected operation. Therefore only the scores for thatoperation (OP11) are averaged, so that the operation LF, OP11, 0.75 isprovided in the composite recommendation set.

The disclosed voting ensembling techniques provides several advantages.One advantage is that this approach exploits the scores to provide moreaccurate estimates.

FIG. 9 illustrates a preferential ensembling process 900 according tosome embodiments of the disclosed technology. The process 900 may beperformed, for example, by the vehicle repair estimating systems 100 and300 of FIGS. 1 and 3 . Referring to FIG. 9 , the process 900 may includedistributing the claim package to the evaluating AI functions, at 902.In the example of FIG. 3 , ensembling AI function 302 may distributeclaim packages 310A,B,C to the evaluating AI functions 304A,B,C,respectively.

Referring again to FIG. 9 , the process 900 may include receiving, fromeach source of a plurality of the sources, a respective vehicle repairrecommendation set for a damaged vehicle, wherein each vehicle repairrecommendation set identifies a recommended vehicle repair operation ofa plurality of the vehicle repair operations for the damaged vehicle, at904. In the example of FIG. 3 , the ensembling AI function 302 mayreceive recommendation sets 312A,B,C from three sources, namely theevaluating AI functions 304A,B,C, respectively.

Referring again to FIG. 9 , the process 900 may include determining arespective source rank of each source from a plurality of the sourceranks, at 906. In the example of FIG. 3 , the evaluating AI functions304 may be ranked. The ranks may be set, for example, by the claimsmanagement system in accordance with preferences for the evaluating AIfunctions 304.

Referring again to FIG. 9 , the process 900 may include generating acomposite vehicle repair recommendation set that identifies therecommended vehicle repair operations in an order determined accordingto the source ranks of the respective sources, at 908. In the example ofFIG. 3 , the ensembling AI function 302 may include, in the compositevehicle repair recommendation set 314, the recommended vehicle repairoperations generated by the evaluating AI functions 304 having thehighest rank, as well as those generated by one or more other evaluatingAI functions, in rank order.

Referring again to FIG. 9 , the process 900 may include providing thecomposite vehicle repair recommendation set to one or more vehiclerepair insurance claims management systems, at 910. In the example ofFIG. 3 , the ensembling AI function 302 may provide the compositerecommendation set 314 to one or more claims management systems 308.

Referring again to FIG. 9 , the process 900 may include determiningwhether a predetermined retraining trigger event has occurred, at 912.The process 900 may include requesting one or more of the evaluatingartificial intelligence functions be re-trained when a predeterminedevent occurs, at 914. In the example of FIG. 3 , the ensembling AIfunction 302 may determine whether a predetermined retraining triggerevent has occurred, and may request one or more of the evaluatingartificial intelligence functions 304 be re-trained when thepredetermined event occurs.

FIG. 10 illustrates an example operation 1000 according to thepreferential ensembling process 900 of FIG. 9 . Referring to FIG. 10 ,the ensembling AI function 1002 receives a claim package from a claimsmanagement system 1008 via an API GW 1006. For clarity, only the imagesin the claim package are discussed. However, it should be appreciatedthat the claim package may contain other data objects in addition to, orinstead of, the images, and that those data objects may be treated in asimilar manner.

The claim package includes five images 1,2,3,4,5 of a damaged vehicle.The ensembling AI function 1002 distributes the images to the evaluatingAI functions 1004. In this example, the ensembling AI function 1002distributes all of the images to each of the evaluating AI functions1004. The evaluating AI functions 1004 process the images.

Each of the evaluating AI functions 1004 returns a differentrecommendation set. The evaluating AI function 1004A returns arecommendation set that includes images 1,2 and two recommendedoperations with components and scores. The legend in FIG. 10 indicatesidentifiers for components and operations. One of the recommendedoperations is LFB, OP11, 0.7, which indicates replacement of the lowerfront bumper with a score of 0.7. Another recommended operation is UFB,OP9, 0.5, which indicates repair of the upper front bumper with a scoreof 0.5.

The evaluating AI function 1004B returns a recommendation set thatincludes images 3,4 and one recommended operation with component andscore. The recommended operation is LF, OP9, 0.6, which indicates arepair of the left fender with a score of 0.6.

The evaluating AI function 1004C returns a recommendation set thatincludes images 2,3,4 and two recommended operations with components andscores. One of the recommended operations is LQP, OP9, 0.5, whichindicates repair of the left quarter panel with a score of 0.5. Theother recommended operation is H, OP26, 0.8, which indicates paintlessdent repair of the hood with a score of 0.8.

As shown in FIG. 10 , the evaluating AI functions 1004A,B,C have ranks2, 3, and 1, respectively. Therefore the ensembling AI function 1002places the recommended repair operations generated by evaluating AIfunction 1004C first in the composite recommendation set, and places therecommended repair operations generated by evaluating AI function 1004Asecond. In some embodiments, the ensembling AI function 1002 may omitthe recommended repair operations generated by evaluating AI functions1004 having lower ranks from the composite recommendation set. In theexample of FIG. 8 , the recommended repair operations generated by thethird-ranked evaluating AI function 1004B have been omitted from thecomposite recommendation set. In other embodiments, all of therecommended repair operations may be included in the compositerecommendation set. In some embodiments, only the recommended repairoperations generated by the highest-ranked evaluating AI function 1004Cmay be included in the composite recommendation set.

The disclosed preferential ensembling techniques provides severaladvantages. One advantage is that this approach allows the recommendedvehicle repair sets in the composite vehicle repair recommendation setto be ranked. These ranks can be set internally, or by externalentities, for example such as a claims management system.

FIGS. 11A, B illustrate a random ensembling process 1100 according tosome embodiments of the disclosed technology. The process 1100 may beperformed, for example, by the vehicle repair estimating systems 100 and300 of FIGS. 1 and 3 . Referring to FIG. 11A, the process 1100 mayinclude receiving a vehicle damage object for a damaged vehicle, whereinthe vehicle damage object includes a plurality of images of the damagedvehicle, at 1102. In the example of FIG. 3 , the ensembling AI function302 may receive a claim package 310 containing the vehicle damage objectfrom a claims management system 308.

Referring to FIG. 11A, the process 1100 may include fragmenting theobject into a plurality of vehicle damage object fragments, wherein eachvehicle damage object fragment includes at least one of the images ofthe damaged vehicle, at 1104. In some embodiments, the object may befragmented into vehicle damage object fragments in a random manner. Inthe example of FIG. 3 , the ensembling AI function 302 may fragment theobject.

Referring to FIG. 11A, the process 1100 may include providing each ofthe vehicle damage object fragments to a respective one of a pluralityof artificial intelligence functions, at 1106. In the example of FIG. 3, ensembling AI function 302 may provide the vehicle damage objectfragments to the evaluating AI functions 304. The ensembling AI function302 may provide the vehicle damage object fragments to the evaluating AIfunctions 304 in a random manner. For example, ensembling AI function302 may generate random associations between the vehicle damage objectfragments and the artificial intelligence functions 304, and may providethe vehicle damage object fragments to the artificial intelligencefunctions 304 according to the generated random associations.

Referring again to FIG. 11A, the process 1100 may include receiving arespective vehicle repair recommendation set from each of the artificialintelligence functions, wherein each vehicle repair recommendation setis based on a respective one of the vehicle damage object fragments, andwherein each of the vehicle repair recommendation sets identifies arecommended vehicle repair operation of a plurality of the vehiclerepair operations for the damaged vehicle, 1108. In the example of FIG.3 , the ensembling AI function 302 may receive recommendation sets312A,B,C from the evaluating AI functions 304A,B,C, respectively.

Referring again to FIG. 11B, the process 1100 may include selecting aplurality of the recommended vehicle repair operations, at 1110. In theexample of FIG. 3 , the ensembling AI function 302 may select therecommended vehicle repair operations returned by the evaluating AIfunctions 304. In some embodiments, the ensembling AI function 302 mayselect the recommended vehicle repair operations in a random manner.

Referring again to FIG. 11B, the process 1100 may include generating acomposite vehicle repair recommendation set that identifies the selectedrecommended vehicle repair operations, at 1112. In the example of FIG. 3, the ensembling AI function 302 may include the selected recommendedvehicle repair operations in the composite vehicle repair recommendationset 314.

Referring again to FIG. 11B, the process 1100 may include providing thecomposite vehicle repair recommendation set to one or more vehiclerepair insurance claims management systems, at 1114. In the example ofFIG. 3 , the ensembling AI function 302 may provide the compositerecommendation set 314 to one or more claims management systems 308.

Referring again to FIG. 11B, the process 1100 may include determiningwhether a predetermined retraining trigger event has occurred, at 1116.The process 1100 may include requesting one or more of the evaluatingartificial intelligence functions be re-trained when a predeterminedevent occurs, at 1118. In the example of FIG. 3 , the ensembling AIfunction 302 may determine whether a predetermined retraining triggerevent has occurred, and may request one or more of the evaluatingartificial intelligence functions 304 be re-trained when thepredetermined event occurs.

FIG. 12 illustrates an example operation 1200 according to the randomensembling process 1100 of FIG. 11 . Referring to FIG. 12 , theensembling AI function 1202 receives a claim package from a claimsmanagement system 1208 via an API GW 1206. For clarity, only the imagesin the claim package are discussed. However, it should be appreciatedthat the claim package may contain other data objects in addition to, orinstead of, the images, and that those data objects may be treated in asimilar manner.

The claim package includes five images 1,2,3,4,5 of a damaged vehicle.The ensembling AI function 1202 distributes the images to the evaluatingAI functions 1204. In this example, the ensembling AI function 1202distributes the images to the evaluating AI functions 1204 in a randommanner. Referring to FIG. 12 , the ensembling AI function 1202distributes images 1,2,3 to the evaluating AI function 1204A;distributes images 2,3,4 to the evaluating AI function 1204B; anddistributes images 1,2,3,4,5 to the evaluating AI function 1204C. Theevaluating AI functions 1204 process the images.

Each of the evaluating AI functions 1204 returns a differentrecommendation set. The evaluating AI function 1204A returns arecommendation set that includes images 1,2,3 and one recommendedoperation with component and score. The legend in FIG. 12 indicatesidentifiers for components and operations. The recommended operation isLFB, OP11, 0.7, which indicates replacement of the lower front bumperwith a score of 0.7.

The evaluating AI function 1204B returns a recommendation set thatincludes images 2,3,4 and one recommended operation with component andscore. The recommended operation is LF, OP9, 0.6, which indicates arepair of the left fender with a score of 0.6.

The evaluating AI function 1204C returns a recommendation set thatincludes images 1,2,3,4,5 and two recommended operations with componentsand scores. One of the recommended operations is FB, OP9, 0.5, whichindicates repair of the front bumper with a score of 0.5. The otherrecommended operation is H, OP26, 0.5, which indicates paintless dentrepair of the hood with a score of 0.5.

The ensembling AI function 1202 may aggregate these recommendation setsto form the composite vehicle repair recommendation set. In thisexample, the composite recommendation set includes a union of the imagesin the recommendation sets provided by the evaluating AI functions 1204,namely images 1,2,3,4,5. The composite recommendation set also includesa union of the recommended operations and scores in the recommendationsets provided by the evaluating AI functions 1204.

The disclosed random ensembling techniques provides several advantages.One or more of the evaluating AI functions 1204 may feature some form ofbias. For example, one or more of the evaluating AI functions 1204 mayprefer repair operations to replacement operations. This bias may beinherent, may be learned over time, or the like. Using the describedrandom techniques helps to mitigate these biases.

FIGS. 13A,B illustrate an adaptive ensembling process 1300 according tosome embodiments of the disclosed technology. The process 1300 may beperformed, for example, by the vehicle repair estimating systems 100 and300 of FIGS. 1 and 3 . Referring to FIG. 13A, the process 1300 mayinclude distributing the claim package to a plurality of sources, at1302. In the example of FIG. 3 , ensembling AI function 302 maydistribute claim packages 310A,B,C to three sources, namely theevaluating AI functions 304A,B,C.

Referring again to FIG. 13A, the process 1300 may include receiving,from each source of the sources, a respective vehicle repairrecommendation set for the damaged vehicle, wherein each vehicle repairrecommendation set identifies a recommended vehicle repair operation ofa plurality of the vehicle repair operations for the damaged vehicle, at1304. In the example of FIG. 3 , the ensembling AI function 302 mayreceive recommendation sets 312A,B,C from three sources, namely theevaluating AI functions 304A,B,C, respectively.

Referring again to FIG. 13A, the process 1300 may include selecting, bya trained artificial intelligence function, one of the recommendedvehicle repair operations for each component of the damaged vehiclebased on a plurality of learned states, at 1306. In the example of FIG.3 , the ensembling AI function 302 may select the recommended vehiclerepair operations returned by the evaluating AI functions 304 based onthe learned states. In some embodiments, the learned states are weights,and each of the evaluating AI functions 304 has a respective weight foreach recommended vehicle repair operation for each component.

Referring again to FIG. 13A, the process 1300 may include generating acomposite vehicle repair recommendation set that identifies the selectedrecommended vehicle repair operation, 1308. In the example of FIG. 3 ,the ensembling AI function 302 may include one or more selectedrecommended vehicle repair operations in the composite vehicle repairrecommendation set 314.

Referring again to FIG. 13A, the process 1300 may include providing thecomposite vehicle repair recommendation set to one or more vehiclerepair insurance claims management systems, at 1310. In the example ofFIG. 3 , the ensembling AI function 302 may provide the compositerecommendation set 314 to one or more claims management systems 308.

Referring again to FIG. 13B, the process 1300 may include determiningwhether the learned states should be adjusted, at 1312. Any techniquemay be used to make this determination. For example, the learned statesmay be adjusted periodically. As another example, the composite vehiclerepair recommendation sets may be compared to baselines, ground truths,and the like. The process 1300 may include retraining the trainedartificial intelligence function by adjusting the learned statesaccording to the vehicle damage objects received, and the correspondingcomposite vehicle repair recommendation generated, since the lastretraining of the trained artificial intelligence function, at 1314. Inthe example of FIG. 3 , the ensembling AI function 302 may determinewhether the learned states should be adjusted, and when it is determinedthe learned states should be adjusted, may retrain using thecorresponding sets of vehicle damage objects received, and compositevehicle repair recommendation sets generated, since the last retraining.

Referring again to FIG. 13B, the process 1300 may include determiningwhether a predetermined retraining trigger event has occurred, at 1316.The process 1300 may include requesting one or more of the evaluatingartificial intelligence functions be re-trained when a predeterminedevent occurs, at 1318. In the example of FIG. 3 , the ensembling AIfunction 302 may determine whether a predetermined retraining triggerevent has occurred, and may request one or more of the evaluatingartificial intelligence functions 304 be re-trained when thepredetermined event occurs.

FIGS. 14 and 15 illustrate example operations before and after adjustingthe learned states according to the adaptive ensembling process 1300 ofFIG. 13 . In these examples, the learned states are weights, asdescribed above.

FIG. 14 illustrates an example operation 1400 at a time T_(t) accordingto the adaptive ensembling process 1300 of FIG. 13 . FIG. 14 includes atimeline, at 1410. The timeline begins at an initial time T₀, where noweights have been learned. Later, at time T_(n), weights have beenlearned. After the weights have been learned, the example operation 1400takes place at T_(t).

Referring to FIG. 14 , the ensembling AI function 1402 receives a claimpackage from a claims management system 1408 via an API GW 1406. Forclarity, only the images in the claim package are discussed. However, itshould be appreciated that the claim package may contain other dataobjects in addition to, or instead of, the images, and that those dataobjects may be treated in a similar manner.

The claim package includes five images 1,2,3,4,5 of a damaged vehicle.The ensembling AI function 1402 distributes the images to the evaluatingAI functions 1404. In this example, the ensembling AI function 1402distributes all of the images to each of the evaluating AI functions1404. The evaluating AI functions 1404 process the images.

Each of the evaluating AI functions 1404 returns a differentrecommendation set. The evaluating AI function 1404A returns arecommendation set that includes images 1,2,3,4,5 and three recommendedoperations with components and scores. The legend in FIG. 14 indicatesidentifiers for components and operations. One of the recommendedoperations is LFB, OP11, 0.7, which indicates replacement of the lowerfront bumper with a score of 0.7. Another recommended operation is UFB,OP9, 0.5, which indicates repair of the upper front bumper with a scoreof 0.5. The remaining recommended operation is LF, OP11, 0.3, whichindicates replacement of the left fender with a score of 0.3.

The evaluating AI function 1404B returns a recommendation set thatincludes images 1,2,3,4,5 and two recommended operations with componentsand scores. One of the recommended operations is H, OP1, 0.8, whichindicates replacement of the hood with a score of 0.8. The otherrecommended operation is LF, OP11, 0.6, which indicates a replacement ofthe left fender with a score of 0.6.

The evaluating AI function 1404C returns a recommendation set thatincludes images 1,2,3,4,5 and three recommended operations withcomponents and scores. One of the recommended operations is FB, OP9,0.5, which indicates repair of the front bumper with a score of 0.5.Another recommended operation is H, OP26, 0.4, which indicates paintlessdent repair of the hood with a score of 0.4. Another recommendedoperation is LF, OP9, 0.7, which indicates a repair of the left fender.

Each of the recommendation sets includes a different recommendedoperation for the left fender, each with a different learned weight. Inthis example, the recommended operation provided by the evaluating AIfunction 1404C has the highest weight, and is selected instead of therecommended operations provided by the other evaluating AI functions1404A,B, as shown in bold font in FIG. 14 .

Two of the recommendation sets include different recommended operationsfor the hood, each with a different learned weight. In this example, therecommended operation provided by the evaluating AI function 1404B hasthe higher weight, and is selected instead of the recommended operationprovided by the evaluating AI function 1404C, as shown in bold font inFIG. 14 .

Two of the recommendation sets include different recommended operationsfor the front bumper. In this example, it is not necessary to considerthe weights of these recommended operations because the recommendedoperations provided by the evaluating AI function 1404A provide greatergranularity, and are selected for that reason instead of the singlerecommended operation provided by the evaluating AI function 1404C, asshown in bold font in FIG. 14 .

Some time after operation 1400, the weights are relearned, at timeT_(t+n+1). FIG. 15 illustrates an example operation 1500 afterrelearning the weights, at a time T_(t+n+2), according to the adaptiveensembling process 1300 of FIG. 13 .

For clarity of explanation, FIG. 15 concerns the same claim package asFIG. 14 . In FIG. 15 , the evaluating AI function 1404C now has a higherweight for the hood operation than the evaluating AI function 1404B, asshown in bold font in FIG. 15 . Therefore the ensembling AI function1402 selects the hood operation provided by the evaluating AI function1404B for inclusion in the composite vehicle repair recommendation set,as shown in FIG. 15 .

The disclosed adaptive ensembling techniques provides severaladvantages. One advantage is that it not only learns and uses thestrengths of the evaluating AI functions, but also relearns thosestrengths over times as the evaluating AI functions change, or arereplaced.

FIG. 16 depicts a block diagram of an example computer system 1600 inwhich embodiments described herein may be implemented. The computersystem 1600 includes a bus 1602 or other communication mechanism forcommunicating information, one or more hardware processors 1604 coupledwith bus 1602 for processing information. Hardware processor(s) 1604 maybe, for example, one or more general purpose microprocessors.

The computer system 1600 also includes a main memory 1606, such as arandom access memory (RAM), cache and/or other dynamic storage devices,coupled to bus 1602 for storing information and instructions to beexecuted by processor 1604. Main memory 1606 also may be used forstoring temporary variables or other intermediate information duringexecution of instructions to be executed by processor 1604. Suchinstructions, when stored in storage media accessible to processor 1604,render computer system 1600 into a special-purpose machine that iscustomized to perform the operations specified in the instructions.

The computer system 1600 further includes a read only memory (ROM) 1608or other static storage device coupled to bus 1602 for storing staticinformation and instructions for processor 1604. A storage device 1610,such as a magnetic disk, optical disk, or USB thumb drive (Flash drive),etc., is provided and coupled to bus 1602 for storing information andinstructions.

The computer system 1600 may be coupled via bus 1602 to a display 1612,such as a liquid crystal display (LCD) (or touch screen), for displayinginformation to a computer user. An input device 1614, includingalphanumeric and other keys, is coupled to bus 1602 for communicatinginformation and command selections to processor 1604. Another type ofuser input device is cursor control 1616, such as a mouse, a trackball,or cursor direction keys for communicating direction information andcommand selections to processor 1604 and for controlling cursor movementon display 1612. In some embodiments, the same direction information andcommand selections as cursor control may be implemented via receivingtouches on a touch screen without a cursor.

The computing system 1600 may include a user interface module toimplement a GUI that may be stored in a mass storage device asexecutable software codes that are executed by the computing device(s).This and other modules may include, by way of example, components, suchas software components, object-oriented software components, classcomponents and task components, processes, functions, attributes,procedures, subroutines, segments of program code, drivers, firmware,microcode, circuitry, data, databases, data structures, tables, arrays,and variables.

In general, the word “component,” “engine,” “system,” “database,” datastore,” and the like, as used herein, can refer to logic embodied inhardware or firmware, or to a collection of software instructions,possibly having entry and exit points, written in a programminglanguage, such as, for example, Java, C, C++, and Python. A softwarecomponent may be compiled and linked into an executable program,installed in a dynamic link library, or may be written in an interpretedprogramming language such as, for example, BASIC, Perl, or Python. Itwill be appreciated that software components may be callable from othercomponents or from themselves, and/or may be invoked in response todetected events or interrupts. Software components configured forexecution on computing devices may be provided on a computer readablemedium, such as a compact disc, digital video disc, flash drive,magnetic disc, or any other tangible medium, or as a digital download(and may be originally stored in a compressed or installable format thatrequires installation, decompression or decryption prior to execution).Such software code may be stored, partially or fully, on a memory deviceof the executing computing device, for execution by the computingdevice. Software instructions may be embedded in firmware, such as anEPROM. It will be further appreciated that hardware components may becomprised of connected logic units, such as gates and flip-flops, and/ormay be comprised of programmable units, such as programmable gate arraysor processors.

The computer system 1600 may implement the techniques described hereinusing customized hard-wired logic, one or more ASICs or FPGAs, firmwareand/or program logic which in combination with the computer systemcauses or programs computer system 1600 to be a special-purpose machine.According to one embodiment, the techniques herein are performed bycomputer system 1600 in response to processor(s) 1604 executing one ormore sequences of one or more instructions contained in main memory1606. Such instructions may be read into main memory 1606 from anotherstorage medium, such as storage device 1610. Execution of the sequencesof instructions contained in main memory 1606 causes processor(s) 1604to perform the process steps described herein. In alternativeembodiments, hard-wired circuitry may be used in place of or incombination with software instructions.

The term “non-transitory media,” and similar terms, as used hereinrefers to any media that store data and/or instructions that cause amachine to operate in a specific fashion. Such non-transitory media maycomprise non-volatile media and/or volatile media. Non-volatile mediaincludes, for example, optical or magnetic disks, such as storage device1610. Volatile media includes dynamic memory, such as main memory 1606.Common forms of non-transitory media include, for example, a floppydisk, a flexible disk, hard disk, solid state drive, magnetic tape, orany other magnetic data storage medium, a CD-ROM, any other optical datastorage medium, any physical medium with patterns of holes, a RAM, aPROM, and EPROM, a FLASH-EPROM, NVRAM, any other memory chip orcartridge, and networked versions of the same.

Non-transitory media is distinct from but may be used in conjunctionwith transmission media. Transmission media participates in transferringinformation between non-transitory media. For example, transmissionmedia includes coaxial cables, copper wire and fiber optics, includingthe wires that comprise bus 1602. Transmission media can also take theform of acoustic or light waves, such as those generated duringradio-wave and infra-red data communications.

The computer system 1600 also includes a communication interface 1618coupled to bus 1602. Network interface 1618 provides a two-way datacommunication coupling to one or more network links that are connectedto one or more local networks. For example, communication interface 1618may be an integrated services digital network (ISDN) card, cable modem,satellite modem, or a modem to provide a data communication connectionto a corresponding type of telephone line. As another example, networkinterface 1618 may be a local area network (LAN) card to provide a datacommunication connection to a compatible LAN (or a WAN component tocommunicate with a WAN). Wireless links may also be implemented. In anysuch implementation, network interface 1618 sends and receiveselectrical, electromagnetic or optical signals that carry digital datastreams representing various types of information.

A network link typically provides data communication through one or morenetworks to other data devices. For example, a network link may providea connection through local network to a host computer or to dataequipment operated by an Internet Service Provider (ISP). The ISP inturn provides data communication services through the world wide packetdata communication network now commonly referred to as the “Internet.”Local network and Internet both use electrical, electromagnetic oroptical signals that carry digital data streams. The signals through thevarious networks and the signals on network link and throughcommunication interface 1618, which carry the digital data to and fromcomputer system 1600, are example forms of transmission media.

The computer system 1600 can send messages and receive data, includingprogram code, through the network(s), network link and communicationinterface 1618. In the Internet example, a server might transmit arequested code for an application program through the Internet, the ISP,the local network and the communication interface 1618.

The received code may be executed by processor 1604 as it is received,and/or stored in storage device 1610, or other non-volatile storage forlater execution.

Each of the processes, methods, and algorithms described in thepreceding sections may be embodied in, and fully or partially automatedby, code components executed by one or more computer systems or computerprocessors comprising computer hardware. The one or more computersystems or computer processors may also operate to support performanceof the relevant operations in a “cloud computing” environment or as a“software as a service” (SaaS). The processes and algorithms may beimplemented partially or wholly in application-specific circuitry. Thevarious features and processes described above may be used independentlyof one another, or may be combined in various ways. Differentcombinations and sub-combinations are intended to fall within the scopeof this disclosure, and certain method or process blocks may be omittedin some implementations. The methods and processes described herein arealso not limited to any particular sequence, and the blocks or statesrelating thereto can be performed in other sequences that areappropriate, or may be performed in parallel, or in some other manner.Blocks or states may be added to or removed from the disclosed exampleembodiments. The performance of certain of the operations or processesmay be distributed among computer systems or computers processors, notonly residing within a single machine, but deployed across a number ofmachines.

As used herein, a circuit might be implemented utilizing any form ofhardware, or a combination of hardware and software. For example, one ormore processors, controllers, ASICs, PLAs, PALs, CPLDs, FPGAs, logicalcomponents, software routines or other mechanisms might be implementedto make up a circuit. In implementation, the various circuits describedherein might be implemented as discrete circuits or the functions andfeatures described can be shared in part or in total among one or morecircuits. Even though various features or elements of functionality maybe individually described or claimed as separate circuits, thesefeatures and functionality can be shared among one or more commoncircuits, and such description shall not require or imply that separatecircuits are required to implement such features or functionality. Wherea circuit is implemented in whole or in part using software, suchsoftware can be implemented to operate with a computing or processingsystem capable of carrying out the functionality described with respectthereto, such as computer system 1600.

As used herein, the term “or” may be construed in either an inclusive orexclusive sense. Moreover, the description of resources, operations, orstructures in the singular shall not be read to exclude the plural.Conditional language, such as, among others, “can,” “could,” “might,” or“may,” unless specifically stated otherwise, or otherwise understoodwithin the context as used, is generally intended to convey that certainembodiments include, while other embodiments do not include, certainfeatures, elements and/or steps.

Terms and phrases used in this document, and variations thereof, unlessotherwise expressly stated, should be construed as open ended as opposedto limiting. Adjectives such as “conventional,” “traditional,” “normal,”“standard,” “known,” and terms of similar meaning should not beconstrued as limiting the item described to a given time period or to anitem available as of a given time, but instead should be read toencompass conventional, traditional, normal, or standard technologiesthat may be available or known now or at any time in the future. Thepresence of broadening words and phrases such as “one or more,” “atleast,” “but not limited to” or other like phrases in some instancesshall not be read to mean that the narrower case is intended or requiredin instances where such broadening phrases may be absent.

What is claimed is:
 1. A system, comprising: a hardware processor; and anon-transitory machine-readable storage medium encoded with instructionsexecutable by the hardware processor to perform a method comprising:receiving a plurality of vehicle repair recommendation sets for adamaged vehicle, wherein each of the vehicle repair recommendation setsidentifies at least one recommended vehicle repair operation of aplurality of the vehicle repair operations for the damaged vehicle,wherein each vehicle repair recommendation set identifies a score foreach of the recommended vehicle repair operations in the plurality ofvehicle repair recommendation sets, wherein each score indicates aprojected accuracy of the corresponding recommended vehicle repairoperation, wherein each of the vehicle repair recommendation setsincludes one or more images of the damaged vehicle, and wherein each ofthe vehicle repair recommendation sets is generated by a respectivefirst artificial intelligence function, and wherein each of therespective first artificial intelligence functions is trained; selectinga plurality of the recommended vehicle repair operations, including atleast one recommended vehicle repair operation from each of the vehiclerepair recommendation sets, by providing the plurality of vehicle repairrecommendation sets to a second artificial intelligence function,wherein the second artificial intelligence function is trained using aplurality of vehicle repair training sets, wherein each vehicle repairtraining set comprises: one or more images of a further damaged vehicle,and a composite vehicle repair recommendation set for the furtherdamaged vehicle; aggregating the selected plurality of the recommendedvehicle repair operations; generating a composite vehicle repairrecommendation set that identifies the aggregated recommended vehiclerepair operations, comprising identifying the scores for the recommendedvehicle repair operations in the generated composite vehicle repairrecommendation set; providing the composite vehicle repairrecommendation set to one or more vehicle repair insurance claimsmanagement systems; and re-training one or more of the first artificialintelligence functions when a predetermined event occurs; wherein: eachof the vehicle repair recommendation sets identifies a plurality ofimages of the damaged vehicle; and the method further comprises:selecting one or more of the images of the damaged vehicle, andidentifying the selected one or more of the images in the generatedcomposite vehicle repair recommendation set.
 2. The system of claim 1,wherein aggregating a plurality of the recommended vehicle repairoperations comprises: aggregating the plurality of the recommendedvehicle repair operations based on statistical aggregation methodologiescomprising at least one of mean, max, min, variance, and standarddeviation.
 3. A non-transitory machine-readable storage medium encodedwith instructions executable by a hardware processor of a computingcomponent, the machine- readable storage medium comprising instructionsto cause the hardware processor to perform a method comprising:receiving a plurality of vehicle repair recommendation sets for adamaged vehicle, wherein each of the vehicle repair recommendation setsidentifies at least one recommended vehicle repair operation of aplurality of the vehicle repair operations for the damaged vehicle,wherein each vehicle repair recommendation set identifies a score foreach of the recommended vehicle repair operations in the plurality ofvehicle repair recommendation sets, wherein each score indicates aprojected accuracy of the corresponding recommended vehicle repairoperation, wherein each of the vehicle repair recommendation setsincludes one or more images of the damaged vehicle, and wherein each ofthe vehicle repair recommendation sets is generated by a respectivefirst artificial intelligence function, and wherein each of therespective first artificial intelligence functions is trained; selectinga plurality of the recommended vehicle repair operations, including atleast one recommended vehicle repair operation from each of the vehiclerepair recommendation sets, by providing the plurality of vehicle repairrecommendation sets to a second artificial intelligence function,wherein the second artificial intelligence function is trained using aplurality of vehicle repair training sets, wherein each vehicle repairtraining set comprises: one or more images of a further damaged vehicle,and a composite vehicle repair recommendation set for the furtherdamaged vehicle; aggregating the selected plurality of the recommendedvehicle repair operations; generating a composite vehicle repairrecommendation set that identifies the aggregated recommended vehiclerepair operations, comprising identifying the scores for the recommendedvehicle repair operations in the generated composite vehicle repairrecommendation set; providing the composite vehicle repairrecommendation set to one or more vehicle repair insurance claimsmanagement systems; and re-training one or more of the first artificialintelligence functions when a predetermined event occurs; wherein: eachof the vehicle repair recommendation sets identifies a plurality ofimages of the damaged vehicle; and the method further comprises:selecting one or more of the images of the damaged vehicle, andidentifying the selected one or more of the images in the generatedcomposite vehicle repair recommendation set.
 4. The non-transitorymachine-readable storage medium of claim 3, wherein aggregating aplurality of the recommended vehicle repair operations comprises:aggregating the plurality of the recommended vehicle repair operationsbased on statistical aggregation methodologies comprising at least oneof mean, max, min, variance, and standard deviation.
 5. A methodcomprising: receiving a plurality of vehicle repair recommendation setsfor a damaged vehicle, wherein each of the vehicle repair recommendationsets identifies at least one recommended vehicle repair operation of aplurality of the vehicle repair operations for the damaged vehicle,wherein each vehicle repair recommendation set identifies a score foreach of the recommended vehicle repair operations in the plurality ofvehicle repair recommendation sets, wherein each score indicates aprojected accuracy of the corresponding recommended vehicle repairoperation, wherein each of the vehicle repair recommendation setsincludes one or more images of the damaged vehicle, and wherein each ofthe vehicle repair recommendation sets is generated by a respectivefirst artificial intelligence function, and wherein each of therespective first artificial intelligence functions is trained; selectinga plurality of the recommended vehicle repair operations, including atleast one recommended vehicle repair operation from each of the vehiclerepair recommendation sets, by providing the plurality of vehicle repairrecommendation sets to a second artificial intelligence function,wherein the second artificial intelligence function is trained using aplurality of vehicle repair training sets, wherein each vehicle repairtraining set comprises: one or more images of a further damaged vehicle,and a composite vehicle repair recommendation set for the furtherdamaged vehicle; aggregating the selected plurality of the recommendedvehicle repair operations; generating a composite vehicle repairrecommendation set that identifies the aggregated recommended vehiclerepair operations, comprising identifying the scores for the recommendedvehicle repair operations in the generated composite vehicle repairrecommendation set; providing the composite vehicle repairrecommendation set to one or more vehicle repair insurance claimsmanagement systems; and re-training one or more of the first artificialintelligence functions when a predetermined event occurs; wherein: eachof the vehicle repair recommendation sets identifies a plurality ofimages of the damaged vehicle; and the method further comprises:selecting one or more of the images of the damaged vehicle, andidentifying the selected one or more of the images in the generatedcomposite vehicle repair recommendation set.